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Abstract 

In this work, we obtain sufficient conditions for tlie "stability" of our recently proposed algorithms, modified-CS 
(for noisy measurements) and Least Squares CS-residual (LS-CS), designed for recursive reconstruction of sparse 
signal sequences from noisy measurements. By "stability" we mean that the number of misses from the current 
support estimate and the number of extras in it remain bounded by a time-invariant value at all times. The concept 
is meaningful only if the bound is small compared to the current signal support size. A direct corollary is that the 
reconstruction errors are also bounded by a time-invariant and small value. 

I. Introduction 

In this work, we study the "stability" of modified-CS (noisy) H], E) and of LS-CS-residual (LS-CS) H, B, 
Is) which were designed for recursive reconstruction of sparse signal sequences from noisy measurements. By 
"stability" we mean that the number of misses from the current support estimate and the number of extras in 
it remain bounded by a time-invariant value at all times. The concept is meaningful only if the bound is small 
compared to the current signal support size. A direct corollary is that the reconstruction errors are also bounded by 
a time-invariant and small value. 

The key assumption that our algorithms utilize is that the support changes slowly over time. As we demonstrated 
in (|3|, H], this assumption holds for many medical image sequences. Denote the support estimate from the previous 
time by T. Modified-CS tries to finds a signal that is sparsest outside of T and satisfies the data constraint. LS-CS 
uses a different approach. It replaces compressive sensing (CS) on the observation by CS on the least squares (LS) 
residual computed using T. Both algorithms are able to achieve greatly reduced reconstruction error compared to 
simple CS (CS at each time separately) when using fewer measurements than what CS needs. 

Other algorithms for recursive reconstruction include our older work on Kalman filtered CS-residual (KF-CS) H, 
lH]; CS for time-varying signals 161 (assumes a time-invariant support, which is a somewhat restrictive assumption); 
homotopy methods I?] (use past reconstructions to speed up current optimization but not to improve reconstruction 
error with fewer measurements); and 18] (a recent modification of KF-CS). Two other algorithms that are also 
designed for static CS with partial knowledge of support include 19] and ifTol . The work of 19] proposed an 
approach similar to modified-CS but did not analyze it and also did not show real experiments either The work 
of ifTOil . which appeared in parallel with modified-CS, assumed a probabilistic prior on the support and obtained 
conditions for exact reconstruction. 

To the best of our knowledge, stability of recursive sparse reconstruction algorithms has not been studied in 
any other work except in our older works 15], ||5] for LS-CS and KF-CS respectively. The KF-CS result Q is 
under fairly strong assumptions, e.g. it is for a random walk signal change model with only support additions 
(no removals). The result for LS-CS stability fS] holds under mild assumptions and is for a fairly realistic signal 
change model. The only limitation is that it assumes that support changes occur "every-so-often": every d time 
units, there are Sa support additions and Sa removals. But from testing the slow support change assumption for 
real data (medical image sequences), it has been observed that support changes usually occur at every time, e.g. 
see Fig. 1 of 13. This important case is the focus of the current work. 

In O, we only studied LS-CS (modified-CS was proposed later). But the techniques of l3] can be also used to 
show modified-CS stability for the model of |3l. In this work, we show the stability of both LS-CS and modified-CS 
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and of its improved version, "modified-CS with add-LS-del". We first discuss modified-CS since, from experiments, 
it is known to be a better algorithm. In facts its stability result is also better (holds under weaker assumptions). 

The paper is organized as follows. We give problem definition in Sec. II-AI and we overview our results in Sec. 
II-BI We describe the signal model for proving stability in Sec. HI] We obtain sufficient conditions for the stability 
of modified-CS and discuss the implications in Sec. HIl] We discuss some of its limitations and develop a simple 
modification that uses a better support estimation approach (modified-CS with add-LS-del). This support estimation 
approach is related to the one in ifTTl . ID, ||3|. In Sec. |IV] we show the stability of modified-CS with add-LS-del, 
which is more difficult to do. The result for LS-CS stability is obtained in Sec. |V] Numerical experiments are 
discussed in Sec. [VT] Conclusions are given in Sec. IVIII 

A. Notation and Problem Definition 

The set operations U, n, \ have their usual meanings. denotes the empty set. We use T"^ to denote the 
complement of a set T w.r.t. [1, m] := [1, 2, . . . m], i.e. T'^ := [1, m] \ T. \T\ denotes the cardinality of T. For a 
vector, V, and a set, T, vt denotes the |r| length sub-vector containing the elements of v corresponding to the 
indices in the set T. \\v\\k denotes the £k norm of a vector v. If just \\v\\ is used, it refers to j|w||2- Similarly, for 
a matrix A/, denotes its induced fc-norm, while just refers to ||M||2. M' denotes the transpose of M 

and denotes the Moore-Penrose pseudo-inverse of M (when M is tall, := {M' M)^^ M'). For a fat matrix 

A, At denotes the sub-matrix obtained by extracting the columns of A corresponding to the indices in T. The 
S'-restricted isometry constant |12J, Ss, for an n x to matrix (with < to). A, and the S, S' restricted orthogonality 
constant lfT2l . 6s. s', are as defined in fT2| eq L3] and lfT2l eq L5] respectively. 

We assume the following observation model: 

yt = Axt + wt, \\wt\\ < e (1) 

where Xf is an to length sparse vector with support N-t, yt is the n < m length observation vector at time t and 
Wt is observation noise with \\wt\\ < e. "Support" refers to the set of indices of the nonzero elements of xt- 

Our goal is to recursively estimate xt using yi,. ..yt- By recursively, we mean, use only yt and the estimate 
from t — 1, Xt^i, to compute the estimate at t. 

As we explain in Sec. Hill our algorithm need more measurements at the initial time, t = 0. We use no to denote 
the number of measurements used at t = and we use Aq to denote the corresponding uq x to measurement 
matrix. We use a to denote the support estimation threshold used by modified-CS and we use aadd,ctdei to denote 
the support addition and deletion thresholds used by modified-CS with add-LS-del and by LS-CS. 

We use Xt to denote the final estimate of xt at time t and Nt to denote its support estimate. To keep notation 
simple, we avoid using the subscript t wherever possible. 

Definition 1 (T, A, A^): We use T := Nt-i to denote the support estimate from the previous time. We use 
A := Nt \ T to denote the unknown part of the support at the current time and Ag := T \ Nt to denote the 
"erroneous" part of T. We attach the subscript t to the set, e.g. Tt or Ai, where necessary. 

Definition 2 (T, A, A^): We use T := Nt to denote the final estimate of the current support; A := A^t \ T to 
denote the "misses" in Nt and Ap ■.~T\ Nt to denote the "extras". 
The sets T^dd, Aadd, Ae,add are defined later in Sec. HVl 

If the sets B, C are disjoint, then we just write DU B\C instead of writing (DU B)\C, e.g. Nt = T U A \ Ag. 

We refer to the left (right) hand side of an equation or inequality as LHS (RHS). 

B. Overview of Results 

When measurements are noisy, the reconstruction errors of modified-CS (noisy) and of LS-CS have been bounded 
as a function of |iVt|, |At| and |Ae (| in JS], ifTSl and in |]3] respectively. The bound is small if |Af| and |Ae,t| are 
small enough. But smallness of the support errors. At, Ae,t, depends on the accuracy of the previous reconstruction. 
Thus it can happen that the error bound increases over time, and such a bound is of limited use for a recursive 
reconstruction problem. There is thus a need to obtain conditions under which one can show "stability", i.e. obtain 
a time-invariant bound on the sizes of these support errors. Also, for the result to be meaningful, the support errors' 
bound needs to be small compared to the support size. 

In this work, we study the stability of modified-CS for noisy measurements and its modification, modified-CS 
with add-LS-del, as well as of LS-CS. This is done under a bounded observation noise assumption and for a signal 
model with 
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1) support changes (Sa additions and Sa removals) occurring at every time, t, 

2) magnitude of the newly added coefficients increases gradually, and similarly for decrease before removal, 

3) support size is 5*0 at all times and the signal power is also constant 

Remark 1: The reason we need the bounded noise assumption is as follows. When the noise is unbounded, 
e.g. Gaussian, all error bounds for CS and, similarly, all error bounds for LS-CS or modified-CS hold with "large 
probabiUty" 113, Ull, fBl, ID, H, El. To show stability, we need the error bound for LS-CS or modified-CS 
to hold at all times, < t < oo (this, in turn, is used to ensure that the support gets estimated with bounded error 
at all times). Clearly this is a zero probability event. 

Our results have the following form. For a given number and type of measurements (i.e. for a given measurement 
matrix. A), and for a given noise bound, e, if, 

1) the support estimation threshold(s) is/are large enough, 

2) the support size, Sq, and support change size, Sa are small enough, 

3) the newly added coefficients increase (existing large coefficients decrease) at least at a certain rate, r, and 

4) the initial number of measurements, no, is large enough for accurate initial reconstruction using simple CS, 
then the support errors are bounded by time-invariant values. In particular, we show that [A^t \ iVt| < 2Sa and 
\Nt \ Nt\ =0. Consequently the reconstruction error is also bounded by a small and time-invariant value. 

A key assumption used in designing both modified-CS and LS-CS is that the signal support changes slowly over 
time. As shown in H, HI, this holds for real medical image sequences. For our model, this translates to 5*^ <C 5*0. 

Under the slow support change assumption, clearly, 2Sa ^ Sq, and so the support error bounds are small 
compared to the support size, ^o, making our stability results meaningful. We also compare the conditions on So 
required by our results with those required by the corresponding simple CS error bounds (since simple CS is not 
a recursive approach these also serves as a stability result for simple CS) and argue that our results hold under 
weaker assumptions (allow larger values of Sq). The results for modified-CS, modified-CS (with add-LS-del) and 
LS-CS are also compared. 

II. Signal model for studying stability 

The proposed algorithms do not assume any signal model. But to prove their stability, we need certain assumptions 
on the signal change over time. These are summarized here. 
Signal Model 1: Assume the following. 

1) (addition) At each t > 0, Sa new coefficients get added to the support at magnitude r. Denote this set by At- 

2) (increase) At each t > 0, the magnitude of Sa coefficients which had magnitude {j — l)r at i — 1 increases 
to jr. This occurs for all 2 < j < d. Thus the maximum magnitude reached by any coefficient is i\/ := dr. 

3) (decrease) At each t > 0, the magnitude of Sa coefficients which had magnitude {j + l)r at t — 1 decreases 
to jr. This occurs for all 1 < j < {d — 1). 

4) (removal) At each t > 0, Sa coefficients which had magnitude r at t — 1 get removed from the support 
(magnitude becomes zero). Denote this set by TZt. 

5) (initial time) At t = 0, the support size is 5*0 and it contains 2Sa elements each with magnitude r, 2r, . . .{d~ 
l)r, and {Sq — {2d — 2)Sa) elements with magnitude M. 

Notice that, in the above model, the size and composition of the support at any t is the same as that at t = 0. 
Also, at each t, there are Sa new additions and Sa removals. The new coefficient magnitudes increase gradually at 
rate r and do not increase beyond a maximum value M dr. Similarly for decrease. The support size is always 
5*0 and the signal power is always (5*0 — {2d — 2)Sa)M^ + 2Sa J2'j=i J^''"^ ■ 

Signal Model [T] does not specify a particular generative model. An example of a signal model that satisfies the 
above assumptions is the following. At each t, Sa new elements, randomly selected from Nt-i', get added to the 
support at initial magnitude, r, and equally likely sign. Their magnitude keeps increasing gradually, at rate r, for 
a certain amount of time, d, after which it becomes constant at M := dr. The sign does not change. Also, at each 
time, t, Sa randomly selected elements out of the "stable" elements' set (set of elements which have magnitude M 
at t — 1), begin to decrease at rate r and this continues until their magnitude becomes zero, i.e. they get removed 
from the support. This model is specified mathematically in Appendix lAl We use this in our simulations. Another 
possible generative model is: at each time t, randomly select Sa out of the 2Sa current elements with magnitude 
jr and increase them, and decrease the other Sa elements. Do this for all 1 < j < d — 1. 

In practice, different elements may have different magnitude increase rates and different stable magnitudes, but 
to keep notation simple we do not consider that here. Our results can be extended to this case fairly easily. 
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To understand the implications of the assumptions in Signal Model [T] we define the following sets. 
Definition 3: Let 

1) I?t(j) := {i : \xt.i\ — jr, \xt-i.i\ = {j + l)?"} denote the set of elements that decrease from {j + l)r to jr 
at time, t, 

2) It{j) ■= {i ■ \xt,i\ = jr, \xt-i.i\ = U — 1)7"} denote the set of elements that increase from {j — l)r to jr 
at time, t, 

3) St{j) := {i : < \xt^i\ < jr} denote the set of small but nonzero elements, with smallness threshold jr. 

4) Clearly, 

a) the newly added set. At :—It{l), and the newly removed set, TZt := l?t(0). 

b) \It{j)\ = Sa, \Vtij)\ = Sa and \StU)\ = 2(j - l)Sa for all j. 

Consider a 1 < j < d. From the signal model, it is clear that at any time, t, Sa elements enter the small elements' 
set, St{j), from the bottom (set At) and Sa enter from the top (set — 1)). Similarly Sa elements leave St{j) 
from the bottom (set TZt) and Sa from the top (set It{j)). Thus, 

St[j) = St-iU) U {At U Vt{j - 1)) \ {TZt U ItU)) (2) 

Since the sets At:'R.t,'Dt{j — l),It(j) are mutually disjoint, and since TZt C St-i{j) and It{j) ^ St-i{j), thus, 

St-i{j) U A \ 7^t = UZt(j) \ Vt{j - 1) (3) 

We will use this in the proof of the stability result of Sec. |lVl 

III. Stability of modified-CS 

Modified-CS was first introduced in Ul as a solution to the problem of sparse reconstruction with partial, and 
possibly erroneous, knowledge of the support. Denote this "known" support by T. Modified-CS tries to find a signal 
that is sparsest outside of the set T among all signals satisfying the data constraint. For recursively reconstructing 
a time sequence of sparse signals, we use the support estimate from the previous time, Nt-i as the set T. At the 
initial time, < = 0, we let T be the empty set, i.e. we do simple CsQ. Thus at i = we need more measurements, 
no > n. Denote the uq x m measurement matrix used at i = by Aq. 

We summarize the modified-CS algorithm in Algorithm [T] Here a denotes the support estimation threshold. 

Algorithm 1 Modified-CS 

For i > 0, do 

1) Simple CS. If t = 0, set T = and compute xt^modcs as the solution of 

mm||(/3)i|i s.t. ||j/o- Ao/3|| < e (4) 

2) Modified-CS. If t > 0, set T = Nt-i and compute Xt.modcs as the solution of 

mm||(/3)To||i s.t. \\yt~Ap\\<e (5) 

3) Estimate the Support. Compute T as 

f = {i e [1, m] : \{xt,modcs)i\ > a} (6) 

4) Set Nt = f. Output Xt,modcs- Feedback Nt. 

By adapting the approach of lfT4l . the error of modified-CS can be bounded as a function of |r| — |iV| + |Ae| — |A| 
and |A|. This was done in lfT3l . We state its modified version here. 

Lemma 1 (modified-CS error bound 4751/ )." If ||u;j| < e and (5|Ar|+|A|+|Ae| < V2 — 1, then 

\\xt - xt^rnodcsW < C\{\N\ + |A| + |Ae|))e, whcrc 



^ ' 1-{V2 + I)5s 



'Alternatively, as explained in [Tl, we can use prior knowledge of the initial signal's support as the set T at 
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If (^|7v|+|A|+|Ae| is just smaller than \/2 — 1, the error bound will be very large because the denominator of Ci{S) 
will be very large. To keep the bound small, we need to assume that it is smaller than fe(\/2 — 1) with a & < 1. 
For simplicity, let & = 1/2. Then we get the following corollary, which we will use in our stability results. 

Corollary 1 (modified-CS error bound): If < e and (5|jv|+|A|+|Ae| < (\/2 ^ then 

\\xt - xt^modcsW < Cii\N\ + \A\ + |Ae|))e < 8.79e (8) 

Proof: Notice that Ci{S) is an increasing function of Ss- The above corollary follows by using (5|Ar|-)-|A|+|Ae| < 
(\/2 — l)/2 to bound Ci(S'). Next, we state a similarly modified version of the result for CS ii4l . 
Corollary 2 (CS error bound l[T4]l ): If ||?z;|| < e and S2\n\ < (\/2 - l)/2, then 

\\xt - xt,cs\\ < Cii2\N\)e < 8.79e (9) 

where Xt^cs is the solution of Q with T = (empty set). 



A. Stability result for modified-CS 

The first step to show stability is to find sufficient conditions for a certain set of large coefficients to definitely 
get detected, and for the elements of Ae to definitely get deleted. These can be obtained using Corollary [T] and the 
following simple facts which we state as a proposition. 

Proposition 1: (simple facts) 

1) An i ^ N will definitely get detected if > a + \\x — XmodcsW- This follows since \\x — XmodcsW > 

11*^ -^modcs II oo ^ I (-^ XjyiQ(j^Qs^-i\. 

2) Similarly, all i G Ag (the zero elements of T) will definitely get deleted if a > ||a; — imodcsH- 
Combining the above facts with Corollary [T] we get the following lemma. 

Lemma 2: Assume that ||ti;|| < e, |Ar| < Sn, |Ae| < 5a,, |A| < Sa- 

1) Let L -.^ {i G N : \xi\ > bi}. All elements of L will get detected at the current time if 6sj^+Sa +Sa < 
(72- l)/2 and fei > a + 8.79e. 

2) There will be no false additions, and all the true removals from the support (the set Ae,t) will get deleted at 
the current time, if Ss^+Sa^+Sa < (^2 — l)/2 and a > 8.79e. 

Notice that in the above lemma and proposition, for ease of notation, we have removed the subscript t. 

We use the above lemma to obtain the stability result as follows. Let us fix a bound on the maximum allowed 
magnitude of a missed coefficient. Suppose we want to ensure that only coefficients with magnitude less than 2r 
are part of the final set of misses. At, at any time, t and that the final set of extras, A^^t is an empty set. In other 
words, we find conditions to ensure that At C St{2) (using Signal Model [T] this will imply that |At| < 25a) and 
|Ae t| = 0. This leads to the following result. The result can be easily generalized to ensure that At C St{do), and 
thus |Af| < {2do — 2)5a, holds at all times t, for some do < d (what we state below is the do ^ 2 case). 

Theorem 1 (Stability of modified-CS): Assume Signal Model [T| and \\wt\\ < e. If the following hold 

1) (support estimation threshold) set a = 8.79e 

2) (support size, support change rate) So,Sa satisfy Ss„+3Sa < {V2 — l)/2 , 

3) (new element increase rate) r > G, where 

G ^ = 8.79e (10) 

2 

4) (initial time) at t ~ 0, jiq is large enough to ensure that At C So{2), \At\ < 2Sa, \Ae.t\ — and iTtl < So 
then, at all t > 0, 

1) |ft| < So, \Ae,t\ = 0, At C St{2) and so |At| < 25^, 

2) |Tt| < So, |Ae,t| < Sa, and |At| < 25a, 

3) II CCt - Xt^rnodcsW < 8.79e 

Proof: The proof follows using induction. We use the induction assumption; Tt — Tt-i', and the fact that 
Nt = Nt-i UAtXTZt to bound |Tt|, |At| and |Ae.t|. Next, we use these bounds and Lemma|2]to bound |Af| and 
|Ae,t|- Finally we use |Tt| < |iVt| + |Ae t| to bound |rt|. The complete proof is given in Appendix IB] 
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B. Discussion 

First notice that condition |4] is not restrictive. It is easy to see that this will hold if the number of measurements 
at t = 0, no, is large enough to ensure that Aq satisfies S2Sa — (v^ — l)/2. 

Clearly, when Sa <C 5*0 (slow support change), the support error bound of 2Sa is small compared to support 
size, 5*0, making it a meaningful stability result. 

Compare the maximum allowed support size 5*0 that is needed for stability of modified-CS with what simple CS 
needs. Since simple CS is not a recursive approach (each time instant is handled separately), Corollary|2l also serves 
as a stability result for simple CS. From Corollary |2] CS needs S2S0 < (\/2 — 1)/2 to ensure that its error is bounded 
by 8.79e for all t. On the other hand, for < > 0, our result from Theorem [T] only needs 5so+3Sa < (v^ — l)/2 to 
get the same error bound, while, at f = 0, it needs the same condition as CS. Said another way, for a given 5*0, at 
t = 0, we need as many measurements as CS doefl while at t > 0, we can use much fewer measurements, only 
enough to satisfy Ssg+aSa < {V^ — l)/2. When Sa ^ Sq (slow support change), this is clearly much weaker. 

C. Limitations 

We now discuss the limitations of the above result and of modified-CS. First, in Proposition [T] and hence 
everywhere after that we bound the £00 norm of the error by the £2 norm. This is clearly a loose bound and 
results in a loose lower bound on the required threshold a and consequently a larger than required lower bound on 
the minimum required rate of coefficient increase/decrease, r. 

Second, we use a single threshold a for addition and deletion to the support estimate. To ensure deletion of the 
extras, we need a to be large enough. But this means that r needs to be even larger to ensure correct detection 
(and no false deletion) of all but the smallest 2Sa elements. There is another related issue which is not seen in 
the theoretical analysis because we only bound £2 norm of the error, but is actually more important since it affects 
reconstruction itself, not just the sufficient conditions for its stability. This has to do with the fact that xt.modcs is 
a biased estimate of xt- A similar issue for noisy CS, and a possible solution (Gauss-Dantzig selector), was first 
discussed in 11211 . In our context, along T'^, the values of Xt^modcs will be biased towards zero, while along T they 
may be biased away from zero (since there is no constraint on (/3)t)- The bias will be larger when the noise is 
larger. This will create the following problem. The set T contains the set Ae which needs to be deleted. Since the 
estimates along Ae may be biased away from zero, one will need a higher threshold to delete them. But that would 
make detecting new additions more difficult, especially since the estimates along A C T'^ are biased towards zero. 

IV. Stability of Modified-CS with Add-LS-Del 

The last two issues mentioned above in Sec. IIII-CI can be partly addressed by replacing the single support 
estimation step by a support addition step (that uses a smaller threshold), followed by an LS estimation step and 
then a deletion step that thresholds the LS estimate. The addition step threshold needs to be just large enough 
to ensure that the matrix used for LS estimation is well-conditioned. If the threshold is chosen properly and if n 
is large enough, the LS estimate will have smaller error than the modified-CS output. As a result, deletion will 
be more accurate and in many cases one can also use a larger deletion threshold. The addition-LS -deletion idea 
was simultaneously introduced in fTTl (CoSaMP) for a static sparse reconstruction and in our older work 13], H 
(LS-CS and KF-CS) for recursive reconstruction of sparse signal sequences. 

Let aadd denote the addition threshold and let add denote the deletion threshold. We summarize the algorithm 
in Algorithmic 

Definition 4 (Define Tadd, ^add, ^e.add)-' The set Tadd is the set obtained after the support addition step. It is 
defined in ( fTTl ) in Algorithmic We use A^dd := \ ?add to denote the missing elements from T^dd and we use 
Ae^add := Tadd \ Nt to dcuotc the extras. 

A. Stability result for modified-CS with add-LS-del 

The first step to show stability is to find sufficient conditions for (a) a certain set of large coefficients to definitely 
get detected, and (b) to definitely not get falsely deleted, and (c) for the zero coefficients in Tadd to definitely get 
deleted. These can be obtained using Corollary [T] and the following simple facts which we state as a proposition, 
in order to easily refer to them later. 

-This can also be improved if we use prior support knowledge at i = as explained in (T]. 
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Algorithm 2 Modifled-CS with Add-LS-Del 

For i > 0, do 

1) Simple CS. If i = 0, set T = and compute xt,modcs as the solution of 

2) Modified-CS. If i > 0, set T = Nt-i and compute Xt. modes as the solution of of Q. 

3) Additions / LS. Compute T^^d and LS estimate using it: 

Jldd = T U {i e T" : \{xt,modcs)i\ > (Xadd} 
(it,add)T.,aj = AT.,Jyt, (£t,add)T-, ==0 (11) 

4) Deletions / LS. Compute T and LS estimate using it: 

T = Tadd \ {« e Tldd : |(£t,add)i| < adel} 

{xt)f = Af'^Ut, {xt)fa=0 (12) 

5) Set Nt = f. Output Xt- Feedback Nt. 



Proposition 2: (simple facts) 

1) An i G A will definitely get detected if \xi\ > aadd + 1 1 a; — XmodcsW- This follows since ||a; — XmodcsW > 

11*^ -^^modcs 1 1 oo ^ I (-^ Xjjiodcs^i\- 

2) Similarly, an i G T^dd will definitely not get falsely deleted if \xi\ > add + ||(a; — iadd)T„jd||- 

3) All i G Ae,add (the zero elements of Tadd) will definitely get deleted if add > ^ a;add)T„ddll- 

4) Consider LS estimation using known part of support T, i.e. consider the estimate {xls)t — At^tj and 
{xls)t'= = computed from y := Ax + w. Let A = N\T where N is the support of x. If j|it;|| < e and if 
5\T\ < 1/2, then ||(a; — .■el5)t|| < V^e + 26'|t|,|a| ||2:a|1 (instead of S\t\ < 1/2, one can pick any b < 1 and 
the constants in the bound will change appropriately). This bound is derived in 13] equation (15)]. 

Combining the above facts with Corollary [T] we can easily get the following three lemmas. 
Lemma 3 (Detection condition): Assume that < e, |7V| < Sn, |Ae| < Sa^, |A| < Sa- Let Ai := {i G 
A : \xi\ > All elements of Ai will get detected at the current time if (^Sn+Sae+Sa 

< {V2 - l)/2 and 

bi > aadd + 8.79e. 

Proof: The lemma follows from fact [T] of Proposition |2] and Corollary [l] 

Lemma 4 (No false deletion condition): Assume that ||w|| < e, |Tadd| < St and |Aadd| < "^a- For a given bi, 
let Ti := {i G Tadd ^ ^i}- ^11 i G Ti will not get (falsely) deleted at the current time if 83^ < 1/2 and 

bi > add + y2e + '^0St,sJ\xa,J\ 

Proof: This follows directly from fact |2] and fact |4] (applied with T = Tadd and A = Aadd) of Proposition |2l 
Lemma 5 (Deletion condition): Assume that ||w|| < e, |Tadd| < St and |Aadd| < Sa- All elements of Ae_add will 
get deleted if 63^ < 1/2 and add > V^e + 2esj.,sA^'A,J\- 

Proof: This follows directly from fact [3] and fact |4] (applied with T = Tadd and A = Aadd) of Proposition |2l 
Using the above lemmas and the signal model, we can obtain sufficient conditions to ensure that, for some 
do < d, at each time t. At C St{do) (so that |A| < {2do — 2)Sa) and |Ae,t| = 0, i.e. only elements smaller than 
dor may be missed and there are no extras. For notational simplicity, we state the special case below which uses 
do = 2. The general case is given in Appendix ID] 

Theorem 2 (Stability of modified-CS with add-LS-del): Assume Signal Model [T| and \\wt\\ < e. If 

1) (addition and deletion thresholds) 

a) aadd is large enough so that there are at most Sa false additions per unit time, 

b) add = V2e + '2.y/S^eso+2Sa.sJ\ 

2) (support size, support change rate) So,Sa satisfy 

a) Ss,+3S^ < (V2-l)/2, 

b) ^So+2Sa,S„ < ^ 
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3) (new element increase rate) r > niax(G'i, 6*2), where 

^ A Ciadd + 8.79e 




(13) 



1 - '2.\/Ki6sa+2Sa-Sa 

4) (initial time) at t = 0, no is large enough to ensure that At C 5o(2), |A| < 2Sa, \^e,t\ = 0, \ft\ < Sq, 
then, at all t > 0, 

1) \ft\<So, |Ae,t| =0, and |At| <25„ 

2) \Tt\ < So, |Ae,t| < Sa, and |At| < 2^, 

3) iTadd.tl < ^0 + 2Sa, |Ae.add,t| < 2Sa, and |Aadd,t| < Sa 

4) \\xt -xt\\<V2e+ (2(?So,2S„ + l)V^r 

5) \\xt - it,morfcs|| < 8.79e 

Proof: The proof again follows by induction, but is more complicated than that in the previous section, due 
to the support addition and deletion steps. The induction step consists of three parts. First, we use the induction 
assumption; Tt — Tt_i; and the fact that Nt = Nt-i U At\Ti-t to bound \Tt\, |Ae.t|, |At|. This part of the proof is 
the same as that of Theorem [T] The next two parts are different and more complicated. We use the bounds from 
the first part; equation Lemma[3l the limit on the number of false detections; and jTaddl < \N\ + |Ae,add| to 
bound |Aadd,t|, |Ae_add,t|, |7add,t|- Finally, we use the bounds from the second part and Lemmas |4] and |5] to bound 
|At|, |Ae,t|, \Tt\. The complete proof is given in AppendixjC] 



Notice that condition |2b] may become difficult to satisfy as soon as Sa increases, which will happen when the 
problem dimension, m, increases (and consequently Sq increases, even though Sa and 5*0 remain small fractions 
of m, e.g. typically 5*0 ~ 10%m and Sa ~ 0.2%to). The reason we get this condition is because in facts |2] and [3] 
of Proposition |2l and hence also in Lemmas |4] and |5] and the final result, we bound the £00 norm of the LS step 
error, \\{x - iadd)T.,jj lU by the £2 norm, - Xadd)T„jd||- 

This is clearly a loose bound - it holds with equality only when the entire LS step error is concentrated in one 
dimension. In practice, as observed in simulations, this is usually not true. The LS step error is quite spread out since 
the LS step tends to reduce the bias in the estimate. Thus it is not unreasonable to assume that || {x — Xadd)T„dd lloo ^ 
II (x — a;add)T„jd||/A/<5^ (LS Step error is spread out enough to ensure this) at all times. In simulations, we observed 
that when m = 200, So = 20, Sa = 2, r = 3/4, n ^ 59, Wt unif{-c,c) with c = 0.1266, and we used 

Oiadd = c/2, adei = t/I this was true 99.8% of the times. The same was true even when r was reduced to 2/5. When 
we increased the problem size five times to m = 1000, 5o = 100, Sa = 10, n = 295, and all other parameters were 
the same, this was true 93% of the times. In all cases, 100% of the times, (||(a;— iadd)Taddll/'\/^)/||(a^~iadd)T;,dJ|oo < 
0.78. All simulations used a random-Gaussian matrix A. 

With this extra assumption. Lemmas |4] and |5] will get replaced by the following two lemmas. With using these 
new lemmas, condition IZb] will get replaced by 9s„+2Sa-Sa < 1/4 which is an easily satisfiable condition. Moreover, 
this also makes the lower bound on the required value of r (rate of coefficient increase/decrease) smaller 

Lemma 6: Assume that \\w\\ < e, |Tadd| < St and |Aadd| < Sa- Also, assume that ||(a; — .Tadd)T;,dd l|oo < 
\\{x — a;add)Tadd II / (the LS step error is spread out enough). For a given 61, let Ti := {i G Tadd : l^^* I ^ ^i}- 
i <E Ti will not get (falsely) deleted at the current time if Ssj, < 1/2, and 61 > adei + {V2e + 29sT,^SA\\^'^M\\)/y^- 

Lemma 7 (Deletion condition): Assume that \\w\\ < e, |Tadd| < St and |Aadd| < "^a- Also, assume that ||(x — 
iadd)T„dd l|oo < ||(a; — iadd)Tadd II />/^ (the LS Step crror is spread out enough). All elements of Ag, add will get deleted 



By using Lemmas |6] and |7] instead of Lemmas |4] and |5] respectively, and doing everything else exactly as in the 
proof of Theorem |2] we get the following corollary. 

Corollary 3 (Stability of modified-CS with add-LS-del): Assume Signal Model [T] and ||wt|| < e. Let et := {xt — 



B. Discussion 
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2) (support size, support change rate) So,Sa satisfy 

a) <Sso+3S„ < (\/2-l)/2, 

3) (new element increase rate) r > niax(Gi, G2), where 

aadd + 8.79e 



Gi 



2 



A ^ (14) 

VS'a(l - 26'5Q+2S„,Sa) 

4) fini/ia/ time) at t = 0, riQ is large enough to ensure that A C 5o(2), |A| < 2Sa, |Ae| = 0, |T| < Sq, 
then all conclusions of Theorem |2] hold. 

Notice that conditions [Tbl l2bl and [3] (r > G2) are weaker compared to those in Theorem [21 while others are the 
same. But of course we also need the LS error is spread out enough. For large sized problems, condition |2b] of 
Theorem |2] is the stronger condition out of the two conditions that SQ,Sa need to satisfy. On the other hand, in 
this corollary, condition |2a| is the stronger of the two since its RHS is larger (Ssg+sSa > ^So+2Sa,Sa) and its LHS 
is smaller ((a/2 — l)/2 = 0.207 < 1/4). Condition l2al is easy to satisfy even for large sized problems. 

Let us compare our result with the CS result given in Corollary ID It needs S2S0 < (v^ — l)/2 = 0.207 to 
achieve the same error bound as our result. On the other hand, if the LS step error is spread out enough, we only 
need Ssg+ss^ < (\/2 — l)/2 = 0.207 (this is the stronger of the two conditions on ^o, Sa)- When Sa ^ -S'o (slow 
support change), in fact as long as Sa < Sq/3, this is weaker than what CS needs. 

Finally, let us compare this result with that for modified-CS (without add-LS-del) given in Theorem [T] Because 
of add-LS-del, the addition threshold, aadd, can now be much smaller, as long as the number of false adds is smaljj 
If ctadd is close to zero, the value of Gi is almost half that of G, i.e. the minimum required coefficient increase 
rate, r, gets reduced by almost half. Notice that since 0So+2S„ < 1/4, so G2 < {2\/2e)/y/S^ < 4.4e < Gi, i.e. the 
upper bound on G2 is smaller than 4.4e, which is the lower bound on Gi. Thus Gi is what decides the minimum 
allowed value of r. 

Remark 2: In the discussion in this paper we have used the special case stability results where we find conditions 
to ensure that the misses remain below 25a. If we look at the general form of the result, e.g. see Appendix ID] for 
modified-CS with add-LS-del, the rate of coefficient increase decides what support error level the algorithm stabilizes 
to, and this, in turn, decides what conditions on 6 and 8 are needed (in other words, how many measurements, n, 
are needed). For a given n, as r is reduced, the algorithm stabilizes to larger and larger support error levels and 
finally becomes unstable. See Fig. [T] Also, if n is increased, stability can be ensured for smaller r's. 



V. Stability of LS-CS (CS on LS residual) 

LS-CS uses partial knowledge of support in a different way than modified-CS. It first computes an initial LS 
estimate using the known part of the support, T, and then computes the LS observation residual. CS is applied on 
the residual instead of applying it to the observation. Add-LS-del is used for support estimation. We summarize 
the algorithm in Algorithm [3] 

The CS-residual step error can be bounded as follows. The proof follows in exactly the same way as that given 
in 13] where CS is done using Dantzig selector instead of (fTSI l. We use ( fT6l l here to keep the comparison with 
modified-CS easier 



^e.g. in simulations with m = 200, So = 20, Sa = 2, r = 0.4, n = 59, wt unif(-c,c) with c = 0.1266, a^dd = 0.06, 

<^del = ''/2, we were able to use aadd = c/2 = 0.06 and stiU ensure number of false adds less than Sa- 
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Algorithm 3 LS-CS with Add-LS-Del 



For i > 0, do 

1) Simple CS. Do as in Algorithm |2l 

2) CS-residual. 

a) Use T := TVf-i to compute the initial LS estimate, Xt.init, and the LS residual, yt^res, using 

(it,init)T = Ar^yt, (xt,init)T<= = 

yt,res = yt~ Axt^init (15) 

b) Do CS on the LS residual, i.e. solve 

mm|l/3|li s.f. liases- A/311 <e (16) 

and denote its output by $f Compute 

it.CSres := /^t + Xt.init- (17) 

3) Additions / LS. Compute T^dd and LS estimate using it as in Algorithmic Use Xt^csres instead of Xt^modcs- 

4) Deletions / LS. Compute T and LS estimate using it as in Algorithm |2] 

5) Set Nt = f. Output xt- Feedback Nf,. 



Lemma 8 ( CS-residual error bound If ||w|| < e, S2\a\ < (V2- 1)/2 and (5|t| < 1/2, 

\\X - Xcsresll < C\\T\, \A\)e + 0|TMA|C"(|r|, |A|)|ixA|l 

C'i\Tl |A|) ^ Ci(2|A|) + %/2C2(2|A|) 




C"(|T|,|A|)^2C2(2|A|)V|T|, where 



Ci{S) is defined in ©, C2(5) = 2 



1 + {V2-I)6s 
l-{V2 + l)Ss 



(18) 



A. Stability result for LS-CS with Add-LS-Del 

The overall approach is similar to the one discussed in the previous section for modified-CS. The key difference 
is in the detection condition lemma, which we give below. Its proof is given in Appendix |E] 

Lemma 9 (Detection condition for LS-CS): Assume that \\iv\\ < e, |r| < St and |A| < ^a. Let b HxaHoo- 
For a 7 < 1, let Ai {z e A : 76 < \xi\ < b} and let A2 ;= A \ Ai. Assume that |Ai| < Sai and 
llxAall < i^b. All i e Ai will definitely get detected at the current time if 52s& < (\/2 - l)/2, 6sj, < 1/2, 
Os^,s^C"iST,SA) < 7/2(v/^+k), and 

max ; < b 

|A|<Sa 7 - 9s^,ia\C"{St, |A|)(^5Z: + 

where C"(., .), C"(., .) are defined in Lemma |8] 

The stability result then follows in the same fashion as Theorem |2l The only difference is that instead of Lemma 
|3] we apply Lemma|9] with St = So, Sa = 2Sa, = 2r, 7 = 1, Sa^ = Sa and k = (^/25Ir)/(2r) = VSl/V2. 

Theorem 3 (Stability of LS-CS): Assume Signal Model [T] and < e. If 

1) (addition and deletion thresholds) 

a) dadd is large enough so that there are at most Sa false additions per unit time, 

b) adei ^ %/2e + 2y/S^0So+2Sa:Sa'r 

2) (support size, support change rate) So,Sa satisfy 

a) <iV2- l)/2 
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b) <5so+2S„ < 1/2 

3) (new element increase rate) r > niax(G'i, 6*2), where 

aadd + C'{SoA^\)e 



|A|<25/2 - (2 + V2)es,,\A\VS'aC"{So, |A|)^ 



= , , (19) 

4) (initialization) (same condition as in Theorem |2]l 
then, all conclusions of Theorem |2] hold for LS-CS, except the last one, which is replaced by \\xt — XtcsresH < 
max|A|<25jC'(5o, |A|)e + (^?5o,|A|G"(5o, |A|) + l)y2^r]. 



B. Discussion 

Notice that conditions |2c] and |2d] are the most difficult conditions to satisfy as the problem size increases and 
consequently Sa increases. We get condition |2d] because we bound the l^o norm of the detection LS step error by 
its £2 norm which is a loose bound. This can be relaxed in the same fashion as in the previous section by assuming 
that the LS step error is spread out enough (see Corollary [3]). 

Consider condition |2c] We get this because (i) we upper bound the loo norm of the CS-residual step error, 
xt — xt.csres, by its £2 norm and (ii) in the proof of Lemma [H we upper bound the £1 norm of the initial LS 
step error, {xt — it,init)T, by y/\T\ times its £2 norm (this results in the expression for C"(., .) given in Lemma 
|8]l. If one can argue that both the initial LS step error and the CS-residual error are spread out enough, we can 
relax condition l2cl to make it somewhat comparable to that of modified-CS. But even then, Gi will be larger and 
so LS-CS will still require a higher rate of coefficient increase, r, to ensure stability. This is also observed in our 
simulations. See Fig. |l(c)| 

VI. Simulation Results 

We compared modified-CS (mod-cs), modified-CS with Add-LS-Del (mod-cs-add-del), LS-CS and simple CS 
for a few different choices of r. In all cases, we used Signal Model [T] with m = 200, S'o = 20, S'a = 2, n = 59 and 
uu^ ^i-t-d. uniform{—c,c) with c = 0.1266. The specific generative model that we used is specified in Appendix 
IaI and also briefly discussed in Sec. |lll The measurement matrix was random-Gaussian. We averaged over 50 
simulations. In all cases, we set the addition threshold, aadd, to be at the noise level - we set it to c/2. Assuming 
that the LS step after addition gives a fairly accurate estimate of the nonzero values, one can set the deletion 
threshold, add, to a larger value of r/2 and still ensure that there are no (or very few) false deletions. Larger 
deletion threshold ensures that all (or most) of the false additions and removals get deleted. Modified-CS used a 
single threshold, a, somewhere in between aadd and add- We set a = ((c/2) + (r/2))/2. 

In Fig. [H we show two sets of plots: r = 1, d = 3 (M = dr = 3) in |l(a)| and r = 0.5, d — 4 [M = dr = 2) in 
|l(c)| Normalized MSB (NMSE), average number of extras (mean of |7Vt \ A^t| over the 50 simulations) and average 
number of misses (mean of \Nt \ Nt\) are plotted in the left, middle and right columns respectively. Notice that 
when the support size is S'o = 20, n ~ 59 is too small for CS to work and hence in all cases, the NMSE of CS 
was more than 20%. We show its NMSE only in |l(a)| 

When r = 1, all of mod-cs, mod-cs-add-del and Is-cs are stable. Mod-cs-add-del uses a better support estimation 
method (add-LS-del) and thus its extras and misses are both much smaller than those of mod-cs (in this case, it 
is possible that if we experimented with many different threshold choices, mod-cs error could be made smaller). 
As a result its reconstruction error is also stable at a smaller value. In this case, since r is large enough, LS-CS 
(which also uses add-LS-del) has similar error to that of Mod-cs-add-del. When r is reduced to 0.75, it becomes 
too small for LS-CS and so LS-CS becomes unstable. LS-CS stability is discussed in Sec. [V] As we explain there, 
due to the CS-residual step, LS-CS needs a larger r for stability. When r is reduced to 0.5, mod-CS also becomes 
unstable. But mod-cs-add-del is still stable. Mod-CS uses one threshold and hence as explained after Theorem [l] 
it needs a larger r for stability than mod-cs-add-del. Finally if we reduce r to 0.4, all three became unstable. 
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(a) r = 3/3, d = 3, M = dr = 3 




(b) r = 3/4, d = 4:, M = dr = 3 




(c) r = 2/4, d = 4, Af = dr = 2 




(d) r = 2/5, d = 5, A/ = dr = 2 

Fig. 1. Normalized MSE (NMSE), number of extras and number of misses over time for modified-CS, modified-CS with 
add-LS-del and LS-CS. In all cases, NMSE for CS was more than 20% in all cases (plotted only in (a)). 

VII. Conclusions 

We showed the "stability" of modified-CS and its improved version, modified-CS with add-LS-Del, and of LS-CS 
for signal sequence reconstruction, under mild assumptions. By "stability" we mean that the number of misses from 
the current support estimate and the number of extras in it remain bounded by a time-invariant value at all times. 
The result is meaningful when the bound is small compared to the support size. 

Appendix 

A. A generative model for Signal Model Q] 

To help understand the model better (and also to simulate it), we describe here one plausible generative model 
that satisfies its required assumptions. This assumes that every element gets added to the support at magnitude r 
and keeps increasing until it reaches magnitude M. Similarly, every element that began decreasing keeps decreasing 
until it becomes zero and gets removed from the supporfl We keep the signs of the elements the same except when 

Another possible generative model is: select Sa out of the 2Sa current elements with magnitude jr and increase them, and decrease the 
other Sa elements 
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the element first gets added (at that time, can set the sign to ±1 with equal probability). 
To specify the generative model, first define 
Definition 5: Define 

1) Increasing set, Inc* := It{l) U It{'^) ■ ■ - Ultid - 1) 

2) Decreasing set, Dect := X't(l) U Vt{2) • • • U Vtid - 1) 

3) Constant set. Con* {i : \xt,i \ = M}. Clearly Con* = TV* \ (Inc* U Dec*). 
The generative model is as follows. At each t > 0, 

1) Update the magnitudes for elements of the previous increasing, decreasing and constant sets. 



(a;t)inct_i 


[\{xt-l)lnct-i\ ' 


f r] sgn((a;t_i)inct_J 




= [|(xt-l)Dect_i| 


-r] sgn((a;t_i)Dect_i) 


{xt)cont-i 


= (a:t-i)cont_i 





(20) 

where sgn(z) is a vector containing the signs of each element of z. 

2) Select the newly added set, 2t(l) C N^_-^, of size Sa uniformly at random. Similarly select the new set of 
decreasing elements, 'Dt{d — 1) C Cont_i of size Sa uniformly at random. Set their values as; 

{xt)i,{i) = r s 

{xt)vt(d-i) = {d- l)r sgn((.Tt_i)p,_ J (21) 

where s is an S'a x 1 signs' vector in which each element is +1 or —1 with probabiUty 1/2. 

3) Compute: 

It{d) = {ie Inct-i : \{xt-i)^\ = {d - l)r} 

Vt{0) = {z e Dect_i : = r} (22) 

and update the increasing, decreasing and constant sets: 

Inct =lnct_iUlt(l)\Tt(d) 
Dect = Dect„i U A(d - 1) \ 2?t(0) 
Cont ^ Cont_i UZt(d) \Vt{d - 1) 
Nt = Inct U Dect U Cout (23) 



B. Appendix: Proof of Theorem Q] 

We prove the first claim by induction. Using condition |4] of the theorem, the claim holds for t = 0. This proves 
the base case. For the induction step, assume that the claim holds at < — 1, i.e. |Ae,t-i| = 0, |2t-i| < 5*0, and 
At_i C St-i{2) so that |At_i| < 2Sa- Using this assumption we prove that the claim holds at t. In the proof, we 
use the following facts often: (a) 7^t C Nt-i and At C N^_^, (b) Nt = Nt-i U A \ T^t, and (c) if two sets B, C 
are disjoint, then, {D f] B'')\J C = D \J C \ B for any set D. 

We first bound \Tt\, |Ae,t| and |At|. Since Tj = ft-i, so \Tt\ < Sq. Since A^,* = Nt^i\Nt = Nt-i n [{N^^^ f] 
At) U7^t] C Ae,t-i U7^t = 7^t. The last equality follows since |Ae,t_i| = 0. Thus |Ae,t| < |7^t| = S'a. Now 
consider |At|. Notice that At = Nt\Nt-i = (7Vt_ln7V/Lln7^^)U(An7^^n7Vt^_l) = (At_ln7^^)U(A^7V^'Ll) C 
(iSt_i(2) n TZt) U At = 5t_i(2) U v4t \ TZt- The second last equality uses At_i C 5f_i(2). Since TZt is a subset 
of 5t_i(2) and At is disjoint with 5t_i(2), thus |At| < \St-i{2)\ + \At\- |7^t| = 2Sa + Sa - Sa = 2Sa. 

Next we bound |At|, |Ae,t| and |Ti|. Consider the support estimation step. Apply the first claim of Lemma |2] 
with Sn = So, Sa^ = Sa, Sa = 2Sa, and bi = 2r. Since conditions |2] and [3] of the theorem hold, all elements of 
Nt with magnitude equal or greater than 2r will get detected. Thus, At C St{2). Apply the second claim of the 
lemma. Since conditions |2] and [T] hold, all zero elements will get deleted and there will be no false detections, i.e. 
|Ae,t| = 0. Finally using \ft\ < \Nt\ + |Ae,t|, \ft\ < So- 
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C. Appendix: Proof of Theorem |2] 

We prove the first claim of the theorem by induction. Using condition |4] of the theorem, the claim holds for 
t = 0. This proves the base case. For the induction step, assume that the claim holds at t — 1, i.e. |Ae,t_i| = 0, 
|?t-i| < 'S'o, and At_i C 5t_i(2) so that |At_i| < 2Sa- Using the induction assumption, we prove that the claim 
holds at t. In the proof, we will use the following facts often: TZt C Nt-i, At C A^^'Li and Nt = Nt-i U At \ Tit- 
Also, if two sets B, C are disjoint, then, [D n B") U C = Z? U C \ i? for any set D. 

Since Tt - Tt-i, so \Tt\^ Sq. Since Ae,t = Nt-i \Nt^ Nt-i n [(iV^^.i n A') U7^t] C Ae,t_i U7^^ = 7^^. The 
last equality follows since |Ae,t-i| = 0. Thus |Ae.t| < \TZt\ = Sa. Next we bound | At |. Note that At ^ Nt\Nt-i = 

(iVt_ln7V^^_l^7^^)u(An7^^niVf_l) ^ (At_ln7^^)u(AniVt=_l) c (5t_l(2)n7^f)uA = 5t„i(2)uA\7et. 

Since 7^t is a subset of 5t_i (2) and A is disjoint with 5t_i(2), thus |At| < |5t_l(2)| + |A|-|'7^t| = 2Sa + Sa-Sa. 

Consider the detection step. There are at most Sa false detects (from condition [Tall and thus |Ae add,t| < l^e.tl + 
Sa < 25^. Thus iTadd.tl < l-^fl + |Ae^add.t| < "S'o + 25a. Next we bound |Aadd,f|- Using the above discussion, 
Af C 5t_i(2) U At \ 7^t. Using Q for j = 2, the RHS equals 5t(2) Ult(2) \ Vt{l). Apply Lemma |3] with 
Sn = 5*0, = Sa, Sa = 25a, and with bi = 2r (so that Ai C It (2)). Since conditions |2] and |3] of the theorem 
hold, all the undetected elements of If (2) will definitely get detected at time t. Thus Aadd.t ^ 5t(2) \ I?f (1). Since 
Pt(l) C 5t(2), so |Aadd,d < |5t(2)| - |2?t(l)| = Sa- 

Consider the deletion step. Apply Lemma|5]with St = So + 2Sa, Sa = Sa- Since condition I2a1 holds. SsQ+2Sa < 
1/2. Since Aadd.t C 5t(2) \ I?t(l), so ||a;Aajd|| < \fSar. Since condition [Tbl also holds, all elements of Ae,add,t 
will get deleted. Thus |Ae,f| = 0. Thus \ft\ < \Nt\ + \Ae,t\ < So- Next, we bound |At|. Apply LemmaHwith 
5t = 5o + 25a, 5a = Sa, bi = 2r. Since Aadd,t C 5t(2) \ Vtil), so \\xa,J\ < \/S2r. By LemmaS to ensure 
that all elements of Tadd.t with magnitude greater than or equal to bi = 2r do not get falsely deleted, we need 
Sso+2Sa < 1/2 and 2r > add + V2e + 2eso+2Sa,Say/S^r. From condition [lb] adei = V2e + 20So+2S^,Say/S^r. 
Thus, we need Sso+2Sa < 1/2 and 2r > 2{y/2e + 29sa+2Sa,Sa"'/Sa'i')- 5sQ+2Sa < 1/2 holds since condition [2al 
holds. The second condition holds since condition [2b] and condition [3] (r > G2) hold. Thus, we can ensure that all 
elements of Tadd.t with magnitude greater than or equal to bi = 2r do not get falsely deleted. But nothing can be 
said about the elements smaller than 2r. In the worst case At may contain all of these elements, i.e. it may be 
equal to St{2). Thus, At C St{2) and so |At| < 25a. 

This finishes the proof of the first claim. To prove the second and third claims for any t > 0; use the first claim 
for t—1 and the arguments from the paragraphs above to show that the second and third claim hold for t. The fourth 
claim follows directly from the first claim and fact [4] of Proposition [2] (applied with x = xt, T = Tt, A = At). 
The fifth claim follows directly from the second claim and Corollary [T] 

D. Appendix: Generalized version of Theorem [2] 

Theorem 4 (Stability of modified-CS with add-LS-del): Assume Signal Model [1] and HuifH < e. Let et {xt — 
iadd,t)T„jd,f Assume that ||et||oo < ||et||/-\/5^ at all t (the LS step error is spread out enough). If for some 
1 < do < d, 

1) (addition and deletion thresholds) 

a) aadd is large enough so that there are at most / false additions per unit time, 

b) adei = ^J^e + 2k36so+Sa+fM'^^ 

2) (support size, support change rate) So,Sa satisfy 

a) Ss„+sM+k.) < (a/2-1)/2, 

b) Sso+s.+f < 1/2, ^ 

3) (new element increase rate) r > max(G'i, 6*2), where 

„ A ctadd + 8.79e 

^1 ^ 1 

do 

G2 ^ , (24) 

4) (initial time) at t = 0, no is large enough to ensure that A C 5o(c?o)> \ A\ < {2do - 2)Sa, |Ae| = 0, IT] < 5o, 
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where 



fci = max(l, 2do — 2) 



k2 = max(0, 2dQ — 3) 



(25) 



then, at all t > 0, 

1) |r| < So, |Ae| = 0, and A C 5t(do) and so |A| < (2da - 2)Sa, 

2) |T| < So, |Ae| < 5a, and |A| < h, 

3) iTaddl <So+Sa+f, |A,.,dd| < 5a + /, and |A,dd| < ^2 

4) llxt - xtll < V2e + fc3V5:(20So,(2do-2)s„ + 1)^ 

5) ||xt - Xt,„iodcs\\ < Ci{So + Sa + fci)e < 8.79e. 

E. Proof of Lemma \9\ 

From Lemma [8] if H^H < e, S2\a\ < - 1 and 5|t| < 1/2, then j|a; - xcsresW < C"(|T|,|A|)e + 
^'|T|,|A|C"(|r|, |A|)||a;A||. Using the fact that ||xa|| < \/\Ai\b + HxaJI; fact [T] of Proposition |2l and the fact 
that for all i e Ai, \xi\ > jb, we can conclude that all i G Ai will get detected if S2\a\ < (V2— S\t\ < 1/2 
and aadd + C'e + 6'C"i|a:A2l| + OC" y/\Ki\b < 76. Using IjxAall < ^6 and |Ai| < 5ai, this last inequality holds 
if eC" < -i/2{JsZ+ k) and °°f+^'" , < b. Since we only know that ITI < St, |A| < 5a, |Ai| < 5ai 




and II^Aall < we need the above four inequalities to hold for all values of |r|, |A|, |Ai|, HxAall satisfying these 
upper bounds. This leads to the conclusion of the lemma. Notice that the LHS's of all the required inequalities, 
except the last one, are non-decreasing functions of |A|, |r|, |Ai| and thus the lemma just uses their upper bounds. 
The LHS of the last one is non-decreasing in |T|, |Ai|, HxAalL but is not monotonic in |A| (since C"(|r|, |A|) is 
not monotonic in |A|). Hence we explicitly maximize over |A| < 5a- B 
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